Adding web crawler connector #499
Conversation
|
@samkul-swe is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
![recurseml[bot]](https://avatars.githubusercontent.com/in/1054195?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review by RecurseML
🔍 Review performed on 70f3381..ad75f81
✨ No bugs found, your code is sparkling clean
✅ Files analyzed, no issues (31)
• surfsense_backend/alembic/versions/38_add_webcrawler_connector_enum.py
• surfsense_backend/app/agents/researcher/qna_agent/default_prompts.py
• surfsense_backend/app/agents/researcher/utils.py
• surfsense_backend/app/config/__init__.py
• surfsense_backend/app/connectors/webcrawler_connector.py
• surfsense_backend/app/db.py
• surfsense_backend/app/routes/documents_routes.py
• surfsense_backend/app/routes/search_source_connectors_routes.py
• surfsense_backend/app/services/connector_service.py
• surfsense_backend/app/tasks/celery_tasks/connector_tasks.py
• surfsense_backend/app/tasks/celery_tasks/document_tasks.py
• surfsense_backend/app/tasks/celery_tasks/schedule_checker_task.py
• surfsense_backend/app/tasks/connector_indexers/__init__.py
• surfsense_backend/app/tasks/connector_indexers/webcrawler_indexer.py
• surfsense_backend/app/tasks/document_processors/__init__.py
• surfsense_backend/app/tasks/document_processors/url_crawler.py
• surfsense_backend/app/utils/periodic_scheduler.py
• surfsense_backend/app/utils/validators.py
• surfsense_web/app/dashboard/[search_space_id]/connectors/[connector_id]/edit/page.tsx
• surfsense_web/app/dashboard/[search_space_id]/connectors/[connector_id]/page.tsx
• surfsense_web/app/dashboard/[search_space_id]/connectors/add/webcrawler-connector/page.tsx
• surfsense_web/app/dashboard/[search_space_id]/documents/webpage/page.tsx
• surfsense_web/components/dashboard-breadcrumb.tsx
• surfsense_web/components/editConnector/types.ts
• surfsense_web/components/homepage/integrations.tsx
• surfsense_web/components/sources/connector-data.tsx
• surfsense_web/content/docs/docker-installation.mdx
• surfsense_web/contracts/enums/connector.ts
• surfsense_web/contracts/enums/connectorIcons.tsx
• surfsense_web/lib/connectors/utils.ts
• surfsense_web/messages/en.json
There was a problem hiding this comment.
Review by RecurseML
🔍 Review performed on ad75f81..ebea98c
✨ No bugs found, your code is sparkling clean
✅ Files analyzed, no issues (3)
• surfsense_backend/app/agents/researcher/nodes.py
• surfsense_backend/app/utils/validators.py
• surfsense_web/hooks/use-connector-edit-page.ts
2 participants
Description
Added a new Webcrawler Connector that allows users to crawl and index web pages into their search spaces. The connector supports both Firecrawl and AsyncChromiumLoader (free fallback) for flexible web scraping capabilities.
Key Changes:
WebCrawlerConnectorclass for web page crawlingindex_crawled_urlsindexer with date range supportsearch_webcrawlerfunction for retrieving indexed web contentMotivation and Context
Users need the ability to index external web content into their search spaces for comprehensive knowledge retrieval. This connector enables:
FIX #468
Screenshots
API Changes
New Endpoints/Routes:
WEBCRAWLER_CONNECTORtype resultsDatabase Changes:
WEBCRAWLER_CONNECTORtoSearchSourceConnectorTypeenumDocumentType.CRAWLED_URLFIRECRAWL_API_KEY(optional),INITIAL_URLS(optional)Change Type
Testing Performed
Checklist
firecrawl,langchain_community)High-level PR Summary